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ABSTRACT 

This report describes a negotiated evaluation model 
used to examine the effectiveness of 10 regional center cooperatives 
(RCCs) that assist small rural New Mexico school districts in 
implementing provisions of the Individuals with Disabilities 
Education Act. The 9-month evaluation aimed to determine the scope of 
existing RCC services, the distribution of funding to member school 
districts, and the feasibility of RCCs assuming fiscal responsibility 
for program implementation. Evaluation consisted of 12 steps 
involving extensive consultation and negotiation with program 
stakeholders, including representatives from school districts, state 
agencies, school boards, parent organizations, and other groups who 
would be affected by study outcomes. The development phase of the 
evaluation included organization of available data, identification of 
major stakeholders, and input from stakeholders in the development 
and refinement of survey and interview instruments. The 
implementation phase involved visiting RCC sites to gather additional 
data, conduct interviews, and compile and analyze data. The last 
phase involved formulating options and recommendations and giving 
stakeholders the opportunity to disagree with interpretation of the 
findings and to modify recommendations. The negotiated evaluation 
model promoted educational change by involving stakeholders in the 
evaluation process and formulation of recommendations. This model is 
easily replicable and can be adapted to evaluations of programs 
serving various student populations. Contains a figure outlining the 
evaluation process. (LP) 
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Don't Kill the Mesaenger! 

Conducting Program Evaluation and Facilitating 
Change through Negotiated Evaluation 



The New Mexico State Board of Education (SBE) established, 
through Regulation No. 84-6, ten regional center cooperatives 
acrosl the state to provide assistance to and address the needs 
of the smaller/rural local education agencies (LEAs) in the 
- ?mp^emen?ationof the Individuals with Disabilities Education Act 
Q (IDEA) formerly the Education of the Handicapped Act EHA) . SBE 
§ Regulation 84-6 permitted New Mexico leas to form regional center 
cooperatives (RCCs) and submit consolidated applications under 
IDEA through the use of the Joint Powers Agreement Act ( 11-1-1 
to 11-1-7, NMSA, 1978). Leeway was also given RCCs to serve non- 
member agencies and to pool other than IDEA funds though 
subsequent approved, multi-year Joint Powers Agreements (SDE, 
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1990) . 

The governance/management functions of the NM-RCCs were 
releqated to "Councils" whose membership was comprised of tna 
local superintendents or state agency administrators of member 
agencies. In seven of the RCCs an LEA was assigned to serve as 
the fiscal agent for the RCC on behalf of member districts. In 
three RCCs (beginning in the 1990-1391 School Year) the RCC 
Councils were designated as their own Boards of Finance (SDE 
1990, 1990a), but this was a temporary solution. 

statement: of the Problem 

After eight years of operation, the ten RCCs reported that 
the joint powers agreement process was cumbersome and rendered 
the collaborative process unwieldy at best and unworkable at 
worst. Concomitantly, several larger, and very vocal LEAs which 
were not member s of RCCs, wrote a letter of c° ncer ? *° 6 
Education Agency (SEA) criticizing the SEA and complaining that 
the state's discretionary funds under IDEA were primarily 
directed to the smaller districts when larger districts were also 
in need of the funding. Multiple meetings with re P^entatives 
of RCCs and single LEAs did not result in any resolutions. Both 
sides were entrenched in their positions. 

At the urging of the RCCs, the New Mexico Department of 
Education prepared a request for proposal (RFP) for a contractor 
to design/ develop, and conduct a statewide study to determine 
fa) the scope of existing Regional Center Cooperative (JVr'..„ - 
services, (b) distribution of funding and (c) the feasibility of 
RCCs assuming the fiscal responsibilities associated with the 
implementation of RCC activities. 

The RFP identified preferred outcomes of the study to 
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include recommendations for (a) alternatives and potential 
services to be provided by RCCs; (b) maintenance and/or 
modification of existing structure/ organization of RCCs; and (c) 
potential RCC collaborative efforts with other statewide agencies 
[e.g., single and member LEAs, Health and Environment Department 
(HED) and community non-profit training centers (WPTCs), state 
supported schools (SSSs) , and institutions of higher education 
(THE)] in addressing statewide needs (SDE, 1990,). The time-line 
given was originally one year. Final time-line was nine months. 

Development of NM- RCC Study 

In selecting a method for the study in response to the RFP, 
the evaluators sought a process that (a) assessed current and 
anticipated claims and critical issues, (b) resulted in 
alternative responses to the identifies issues, and (c) assured 
participatory decision making. 

Strategic Planning for Non-Prof it Agencies, adapted from 
Dietrich (1986, p. 6) served as the basic framework for the 
study. The Dietrich model offers a process for establishing a 
strategic fit (figure 1) between the non-profit organization 
^RCCs) and the environment (local, state and national governments 
and educational agencies) and; 

1. uses existing or available data 

2. can be accomplished within flexible time frames 

3. is participatory 

4. includes a focus on the existing agency (RCCs) as well as 
the outside world 

5. is creative 

6. results in practical outcomes 

The modified strategic planning process, adapted for the NM- 
RCC Study, require 12 steps within three major phases (Figure 1) . 
The 12 steps offer a structure within which interview and survey 
studies were to be designed and implemented with extensive 
involvement and negotiation with all stakeholders. Cuba and 
Lincoln (1989, p. 40) define "stakeholder" as persons or groups 
that a?e put at some risk by the study. Stakeholder includes: 
(a) agents, (b) beneficiaries, and (c) potential beneficiaries 
such is, persons or groups potentially benefitted or negatively 
affected by (a) the use of RCCs or (b) by a failure of the 
evaluators to include them in the study. 
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Figure l 



Phasea & Steps for Strategic Planning with Fourth Generation Principles 



sup 1 

Inftfete Contract with Sponsor CowaUsIoning Stucn/ 
(Sponsor/Agent) 



r 



Step 2 
Start Up Project 
(Agent) 



I 



Step 3 

Collect end Organile Available Date 
Identify Stakeholders vie Nomination ol Peers 



step 4 

Review Available 4 Identify Heeded Date 
(Agent, Subcontractor* 4 Steering Coanittee) 



t 



Step 5 

Oraft Survey 4 Interview Instrument! 
(Agent 4 Subcont rectors) 



I 



Step 4 

Review Oreft Research Instruments 
with Stakeholder 4 Finalize 
(Agent) 



Step 7 

Conduct Study for New Data 
(Aeent 4 Stakeholders) 



Step 4 

Cowptle Findings end Analyze 
(Agent 4 Subcontractors) 



Step 9 

Foreulate Option* 4 ft ec amends t tor* 
(Agent, Subcontrectora 4 Steering Committee) 
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Step 10 

Negotiate R ec amende t tor* w/ Cost Cetlewjtee 
(Agent, stakeholders 4 Steering Com(ttee) 
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Step 11 

Develop Final Report of Findings 4 ftKoemndsttc 
« A gent 4 3 wtc on tree tor a) 



Step 12 
Close Out Project 
(Sponsor/Agent) 
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The Development Phase included the first six steps of the 
Negotiated Evaluation process. 

Step l ; The evaluator operational i zed the RFP award through an 
agreement with the contracting agency (agent) . 

The contract actually required three months to negotiate 
through state procedures cutting the length of time for the study 
from one year to nine months. 

Step 2 ; Project start-up included appointment of a project 
Steering Committee and selection of expert sub-contractors to 
analyze data. 

In the New Mexico study, the Steering Committee was 
comprised of representatives of stakeholder groups, nominated by 
members of the representative groups (i.e., RCC staff and 
Governing Councils, single non-member LEAs, state agencies using 
RCCs for services, school boards association, administrators 
association, parent organizations, school collaboratives not 
RCCs, legislative education committees, state department of 
education, human services department, universities' distance 
education, governor's education advisor, state IDEA panel). 
Nominations were sought by written request to members of the 
groups as well as through interviews with prominent individuals. 
Representative stakeholder were taken from those receiving the 
most nominations with consideration given to geographic 
representation . 

The representative stakeholders were involved in the 
development of the questions to be asked, the design of the 
research instruments, and the methods to be used to gather the 
data (survey and interview) . 

The selected subcontractors were recognized in the state as 

(a) expert in school finance, and (b) in New Mexico school law. 

Step 3: While nominations were in process, known available data 
was collected from: the local, state and federal level. 

The available data was organized and presented with the 
Steering Committee membership. 

step 4 : The steering Committee was convened (a) to review 
available data and information baselines, and (b) to identify 
additional data needed regarding organization (RCCs) and 
environment (LEAs, State, HED, SSSs, private providers, etc.). 

The Steering Committee and project staff (a) defined key 
characteristics of data needed and potential data sources, and 

(b) identified additional stakeholders not already included in 
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the steering committee. Additional stakeholder were identified 
by the interview question , "Who disagrees with you that might 
bring another point of view into the study?" 

Step 5 : The evaluators developed draft survey and interview 
instruments with a projected time frame for the study and 
identified stakeholder to be involved. 

Step 6: Evaluators reviewed proposed research design and 
instruments with stakeholder groups through statewide meetings 
such as: Special Education Quarterly Meetings hosted by the SEA, 
School Boards Association, School Administrators' Association, 
Developmental Disabilities Planning Council, Interagency Council 
under P.L. 99-457, IDEA State Advisory Panel, and state Council 
for Exceptional Children. 

These contacts resulted in (a) revised data collection 
instruments, (b) proposed timeline for interview visits to 
regions and other stakeholder groups, and (c) proposed mailings 
and return dates for surveys. 

With the completion of Step 6 the study entered the 
implementation phase. 

Implementation of NM-RCC Study. 

The Implementation Phase was one phase in which the major 
responsible parties were the project staff and agency support 
services. 

Step 7 : Activities involved travel to each of the 10 RCCs for 
(a) data gathering; and (b) interviews with Council members, RCC- 
staffs, contractors, beneficiaries, and anticipated 
beneficiaries . 

Because of the small number of RCCs, a full census of all 
RCC directors and Council members was conducted. Stratified 
random samples were used with RCC-staff , district principals and 
staff, parents and other identified beneficiaries and anticipated 
beneficiary groups (e.g., private providers, HED, IHE) . 

Except for RCC descriptive and comparative data 
(demographic, personnel employment, and fiscal) , confidentiality 
was granted all respondents to the questionnaires and interviews. 

D ata Analysis and Reporting 

The Data Analysis and Reporting phase of the study contained 
five major steps. These steps involved project staff and 
subcontractors in Step 8 for the analysis of data, and the 
Steering Committee and identified major stakeholders in the 
prioritization of options (Steps 9 and 10) . Development of the 
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final reports for the contracting agency (Steps 11 and 12) were 
the responsibility of project staff and subcontractors; however, 
both the interpretation of findings and the recommendations were 
negotiated with stakeholders in state meetings prior to 
finalization with the steering committee. 

Step 8 , Analysis of data included descriptive statistics 
(averages, etc.)/ and non-parametrics (e.g., use of chi square to 
identify statistically significant differences between groups) . 
Comparisons of findings were made with national findings from 
three national studies conducted by Fletcher, Cole, and Strumor 
(1988, 1990) and Fletcher (1991). 

The evaluators organized findings into graphs and tables; 
analyzed them for statistically significant differences between 
groups (i.e., members of RCCs and non-members of RCCs; 
Professionals and parents) ; and compared state findings with 
national findings, where applicable, and developed preliminary 
recommendations . 

Step 9 : Evaluators presented findings, analyses, and proposed 
options to Steering Committee, and generated additional options 
and recommendations. 

Step 10 ; Evaluators and Steering Committee members presented the 
committees' recommendations and options to all major stakeholders 
for comments through conferences under the sponsorship of the 
SEA. 

It was at these conferences and meetings that stakeholders 
could disagree with interpretation of findings and offer 
alternative interpretations and modify recommendations. Every 
comment was considered, and stakeholders who had the authority to 
veto any single recommendation with the legislature were 
purposefully sought out, and asked whether or not they could 
support the recommendation (s) . If not, what they could support 
was garnered and taken back to the stakeholder groups for their 
further consideration. In all instances, the stakeholder groups 
supported the negotiated recommendation. 

In this low populated state, talent is hard to come by and 
some very talented individual's were going to be negatively 
affected financially by the recommendations. Meetings with these 
individuals were also carried out so that the alternative 
recommendations could be discussed. With the personal 
consideration to their dilemma's, they assured the stakeholders 
that they could "live with" the recommendations. Their responses 
assured concurrence among groups at the 90% agreement rate. 

Step 11: The negotiated findings and recommendations were 
prepared into a final report with recommendations for each 
critical issue: (a) legal organization — with a name change 
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recommendation to better define the mission; (b) funding issues, 
(c) IDEA provisions, (d) potential services — required and 
optional, and (e) potential costs of each recommendation. 

Step 12 ; Evaluators submitted the final report with an 
Administrative Summary and the nonidentif iable data collected 
during the study to the contracting agent. The evaluators made 
themselves available to discuss the report with governmental 
committees, administrative bodies, and other boards upon request. 
Copies of the Executive Summary were mailed to all stakeholders 
after members of the State IDEA Advisory Panel accepted the 
report and supported the implementation of the recommendations. 

Conclusion 

The application of a Negotiated Evaluation process gave 
stakeholders throughout a geographically large, but sparsely 
populated state an involvement in a evaluation whose outcome 
could impact them either positively or negatively as an 
organization or as an individual. That involvement included the 
development of the questions to be answered, stakeholders to be 
involved from all sides of the issue, and negotiations of the 
interpretation of the findings and ultimate recommendations. 
Rumors were kept to a minimum for all activities occurred in 
public forums. Outcomes were acceptable to 90 percent or more of 
the stakeholders across stratified groups and individuals 
impacted negatively were made aware of the reasons for the 
recommendations and given opportunities to minimize the negative 
effects. 

Potential for change was greatly enhanced by the full 
inclusion of stakeholders. Turf maintenance and political 
ideologies, so prevalent in rural states, were purposefully 
incorporated into the design rather than skirted by the 
evaluators. This negotiated evaluation model provided a system 
for educational program evaluation that is flexible in timeline? 
easily replicable; and can be adapted to address the formative 
and summative evaluations of programs serving populations at- 
risk, children with disabilities, bilingual programs, and other 
emerging issues highly charged with "turf isms." 

This particular Negotiated Evaluation process resulted in 
desired legislation which two years after the study established 
through statute Regional Educational Cooperatives (Article 2B, 
Regional Cooperative Education, NMSA 1976 as amended 1993). 
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